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Assessing Technical Achievement in Secondary CTE: 
Overview oe State Assessment Systems 


States are using a variety of approaches to assess secondary students’ career technical education 
(CTE) skill attainment under Perkins. According to data reported in the U.S. Department of 
Education’s Report to Congress on State Performance, Program Year 2002-03, states and 
territories are currently using the following approaches to collect performance data for the CTE 
skill attainment measure: 

■ National/State/Local Assessment Systems — 30 states 

■ Grade Point Average in Occupational Subjects — 9 states 

■ Program Completion — 8 states 

■ Course Completion — 7 states 

Although it appears that grantees have limited their measurement to these four collection 
strategies, with most using some form of national, state, or local assessment system, in practice 
state accountability systems produce very different types of data, even when controlling for 
measurement approach. Eor example, states using national assessments may tailor exams to 
address state CTE standards, include different students in their population base, or establish 
different performance thresholds to indicate skill attainment. Consequently, there is presently 
little consistency in how states assess CTE skill attainment for Perkins reporting purposes. 

A review of state measurement approaches conducted for this paper indicates that roughly one- 
third of states (36 percent) are using some form of vendor-developed assessment system, or their 
own state -established exam to collect skill attainment data for Perkins reporting purposes. Many 
states are concurrently using state licensing or industry credentialing exams to award skill 
credentials to secondary students, although not all choose to report this information for Perkins 
accountability purposes. 

This paper summarizes three strategies that states are using to quantify students’ CTE s ki ll 
holdings: national assessments systems; state developed occupationally specific, end-of-course or 
program exams; and state-developed, occupationally generic, end-of-program exams. The paper 
also profiles collection systems in a subset of states representative of each assessment approach. 
State profiles detail the (1) primary characteristics of state assessment systems (2) process and 
cost of developing assessments; (3) test administration policies and procedures; and (4) the 
potential portability of these systems to other states. This information is intended to help inform 
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policy discussions that will occur at an upcoming, OVAE-sponsored conference on student CTE 
skill assessment hosted hy the National Research Center for Career and Technical Education. 


Identification of Sources: Data Collection Methodology 

To profile state measurement strategies, MPR researchers reviewed the extent literature on state 
assessment systems. This included conducting weh searches of reports available on the Internet, 
information posted on the wehsite of the National Research Center for Career and Technical 
Education (www.nccte.org), the National Association of State Directors of Career Technical 
Education (www.careertech.org), the National Occupational Competency Testing Institute 
(www.nocti.org), VTECS (www.vtecs.org), WorkKeys (www.act.org/workkeys), and OVAE’s 
Peer Collaborative Resource Network (www.edcountability.net). 

Researchers also conducted on-line searches of resources available within state department of 
education websites for states identified as having developed or implemented standardized 
assessments based on national or state CTE standards. States identified for initial follow-up 
included — Arizona, Arkansas, Connecticut, Idaho, Illinois, Missouri, New Jersey, Pennsylvania, 
North Carolina, Kentucky, New York, Ohio, Oklahoma, Utah, Vermont, Virginia, West Virginia, 
and Wyoming. ^ 

Based on a review of state information, researchers selected a subset of states using innovative 
strategies that were judged illustrative of the differing approaches used to assess student 
performance. These included — Connecticut, Kentucky, North Carolina, Pennsylvania, Utah, 
and Wyoming. Researchers summarized the characteristics of these state systems using a 
common protocol to extract information (see Appendix A). State profiles were shared with 
administrators in participating states, with contacts asked to review the description of their state 
assessment approach and, where necessary, to make any modifications to ensure that the write-up 
accurately captured the mechanics of state testing procedures. 


* While every attempt was made to ensure that states using either a national or state developed, 
standardized assessment were included, it is possible that there are some states were inadvertently 
excluded. 
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State Assessment Approaches 


An analysis of state reporting approaches indicates that roughly one-third of states are using some 
form of standardized national and/or state assessment to collect data for Perkins accountability 
purposes or to award CTE skill certificates in addition to a regular high school diploma. 
Generally, states fell into one of three assessment categories. Some states have opted to use 
national assessments to track student performance, with third-party, occupationally specific tests 
serving as the primary assessment tool. Other states have established their own occupationally 
specific assessments, administered to students following program or course completion. And one 
state has developed a unique, performance -based assessment to document student attainment of 
generic work-readiness skills common to all CTE program areas. 

Since considerable differences were noted in assessment approaches across the group of eighteen 
states selected for follow-up, as well as among states using similar assessment strategies, 
researchers selected a subset of six states that were deemed to be using innovative assessment 
systems. This section briefly describes the assessment approach used in each of the states 
identified for this study, with detailed descriptions provided for the six states selected for 
profiling. 


Strategy 1: National Assessment Systems 

Eour states — Connecticut, New Jersey, Pennsylvania, and Virginia — currently assess student 
skill attainment for Perkins using standardized, end-of-program exams developed by third-party 
vendors, state licensing agencies, or national business and industry associations. A number of 
other states, including Idaho, New York and Vermont, encourage local districts to adopt national 
exams, but have yet to institute a formal statewide assessment system for Perkins reporting 
purposes. Einally, Arizona and Massachusetts are in the process of assessing the feasibility of 
using standardized, vendor-developed exams to structure CTE reporting in advance of the Perkins 
reauthorization. 

Although each of the identified states uses a combination of assessment strategies to measure s ki ll 
attainment, most have contracted with, or rely upon the National Occupational Competency 
Testing Institute (NOCTI) to serve as their primary testing agent. Generally, these states have 
either worked with NOCTI to develop customized, program specific assessments that align with 
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their own CTE performance standards, or have authorized the use of existing NOCTI assessments 
for statewide use. 

In addition to NOCTI, identified states are also using their own credentialing exams that are 
administered hy state licensing agencies. These assessments are typically associated with 
occupations that affect the health or well being of the public, including the health fields, 
barbering, and cosmetology. Standardized credentialing exams created by national industry 
associations are also in use. For example, most states have adopted exams created by the National 
Institute of Machining Skills, Inc. (NIMS) to assess machining and metalworking, and the 
National Automotive Technicians Education Foundation (NATEF) for automotive repair. 

In addition to statewide occupationally specific assessments, some states are using ACT’s 
WorkKeys system to assess student attainment of generic career readiness skills, although no 
state currently employs the test as its statewide Perkins CTE assessment. According to ACT 
representatives, the WorkKeys Applied Mathematics and Reading for Information exams are 
currently used in Illinois, and will soon be introduced in Michigan, to test all secondary students 
as part of a statewide assessment system. WorkKeys exams are also employed in some secondary 
school districts, and more widely at the community college level, to assess students’ general work 
readiness s ki lls. 


About the NOCTI and WorkKeys Exams 

Most identified states rely on NOCTI to develop and administer a relatively larger proportion of 
their CTE assessments. NOCTI exams are occupationally specific, meaning that they are 
designed to assess specific technical skills in a narrow occupational area. The exams are 
developed by teams of business, industry, and education experts who work to distill critical 
competencies from occupational and workplace standards. Academic standards embedded within 
occupations may also be distinguished. Identified skills are used to construct an occupationally 
specific Job Readiness exam, which prior to release, is piloted in schools around the nation to 
assess student performance. This development process is intended to ensure that NOCTI exams 
are valid and reliable measures of occupational content. The organization has profiled 87 Job 
Ready Assessments for use by secondary agencies, nearly all of which assess occupationally 
specific skills associated with a discrete CTE program area (three exams focus on general 
workforce readiness skills). A list of NOCTI exams is included in Appendix B. 
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The NOCTI Job Ready Assessments consist of a written and performance testing component. 
Written exams usually consist of 150 to 200 multiple-choice questions, which may be 
administered either on-line or using traditional paper-and-pencil methods. Students have up to 
three hours to complete their written assessment, although states may tailor exams to provide 
shorter testing periods. Performance exams typically have 3 to 5 simulated tasks that students 
must perform. Performance outcomes are scored by advisory council members or teachers, and 
may be sent to NOCTI for further processing. 

In addition to test development, NOCTI also provides support services to assist states in 
administering and scoring exams and in reporting student outcomes, using both norm and 
criterion-referenced scoring criteria. Test results are compiled and sent back to the state to 
support program improvement efforts and to comply with federal and state accountability 
requirements. 

It appears that NOCTI is expanding its marketing to the field. In an effort to increase business, in 
September 2005 NOCTI launched a testing initiative geared at encouraging states to adopt 
NOCTI exams to support CTE assessment. Specifically, the organization has revamped its pricing 
structure to offer discounted services based on the volume of assessments used, and has increased 
the type of services offered. For example, states will now be able to bank assessment credit from 
year-to-year, have access to a customized statewide electronic newsletter, and be provided with a 
state -branded, online delivery system. 


Although no state is currently using WorkKeys as its Perkins technical skill attainment measure, 
WorkKeys is a unique assessment designed to assess individuals’ general work readiness s ki lls 
across a number of dimensions. Assessments exist in ten content areas, including: 


■ Reading for Information 

■ Applied Mathematics 

■ Business Writing 

■ Writing 

■ Locating Information 


Teamwork 

Observation 

Listening 

Applied Technology 
Readiness (screening assessment) 


Unlike the NOCTI exams, which measure occupationally specific skills in a specific occupation, 
WorkKeys assessments measure basic skills that all workers require to succeed in the workplace. 
For example, the Reading for Information test measures the skills that people need to read and 
use written information as part of their job. Test items require individuals to read and interpret a 
variety of workplace communications, including memos, letters, directions, signs, bulletins, 
policies and workplace regulations. Other tests assess other broad, career-readiness skills. 
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Educators and employers can use WorkKeys to assess whether individuals have the skills needed 
to succeed in the labor market in general, or in a specific occupational area. To do so, WorkKeys 
has profiled the skills required for success in any of 10,000 joh titles, ranging from accountant to 
welder. These skill levels have been cross-referenced with WorkKeys skill levels that an 
individual must have to perform successfully. According to ACT, by comparing occupational job 
profile data with students’ WorkKeys test scores, an educator or employer can reliably predict 
whether an individual has the skills needed for success.^ Although WorkKeys has not, to date, 
achieved as widespread use at the secondary level as NOCTI, it might have the potential to serve 
as a cost-effective approach for assessing the work readiness skills of all students pursuing studies 
in occupations that share a similar base of skills, such as those found within a career cluster area. 

National Assessment State Profiles: Connecticut and Pennsylvania 

To illustrate state approaches using national assessments, this paper profiles assessment systems 
used in Connecticut and Pennsylvania, which have taken slightly differing approaches to 
instituting CTE assessments (see Appendix C for a detailed summary of state systems). In 
particular, Connecticut has partnered with NOCTI to create customized exams that align with 
state -identified CTE performance standards and competencies, as well as nationally recognized 
industry standards. Since educators are required to align their curriculum with state CTE 
standards as a condition for receiving state funding, tailoring NOCTI exams to existing state 
standards has helped ensure that CTE educators are preparing students for assessment success. 

Pennsylvania, in contrast, has opted to assess students’ skill attainment using existing, NOCTI 
Job Readiness Assessments.^ This has required that state educators reevaluate their curriculum to 
ensure that students are being prepared for the test. Since the state had not adopted a set of 
performance standards for CTE — a generic set of standards is currently in development — local 
administrators are encouraged by the state to identify gaps between their local curriculum and 
national standards. 

To help align curriculum, Pennsylvania educators are encouraged to draw on NOCTI resources, 
which include assessment blueprints that have been developed for each of the organization’s Job 


^ Information drawn from ACT’s WorkKeys website: http://www.act.org/workkeys/index.html 
^ During initial development, the state also worked with NOCTI to align CTE assessments in 
approximately 20 high use occupational areas with the Pennsylvania math standards. This was done in an 
attempt to assess the contribution that CTE makes to students’ academic attainment: scores on the state 
academic assessment in the 11* grade were to be compared with scores on students CTE assessment 
administered in the 12* grade. 


7 


Ready Assessments. NOCTI blueprints provide an outline of the assessment instrument, identify 
core competencies and tasks associated with the exam, and provide sample questions and jobs for 
which the assessment would apply. Since a good school curriculum is one based on the critical 
core competencies required within an occupation, NOCTI contends that there should already be a 
high degree of correlation between NOCTI assessment content and secondary curriculum. 

In addition to the pedagogical implications of aligning (or failing to align) state standards with 
assessments, there are also fiscal tradeoffs between the two state approaches. In Connecticut, the 
cost of aligning NOCTI exams with state standards required that the state make an initial 
investment, estimated at $300,000, to align 18 CTE assessments. In contrast, Pennsylvania 
incurred no quantifiable cost in adopting NOCTI, since local districts rely on ‘off-the-shelf 
exams for assessment purposes. However, since educators may have been required to revamp 
their curriculum to comply with NOCTI assessments, it is likely that districts incurred some cost 
in shifting to the new assessment approach. 

Both states have contracted with NOCTI to administer annual exams (either on-line or in paper 
and pencil format), and to score and report assessment outcomes to the state. The per-student cost 
of administering a Job Ready Assessments varies with the type of assessment and manner in 
which it is administered. For the 2005-06 school year, NOCTI charged states $20 for each student 
who was administered an on-line multiple-choice exam and had a performance rating entered, and 
$25 for each student administered a paper/pencil multiple choice exam and a paper/pencil 
performance rating entered. Costs vary with the options selected. According to state 
administrators, Connecticut spent approximately $265,000 and Pennsylvania approximately 
$540,000 in assessment costs in the 2004-05 school year. 


National Assessment: Other State Testing Efforts 

A number of states, including New Jersey and Virginia, are using NOCTI exams in conjunction 
with tests developed by national business and industry groups. For example. New Jersey 
recognizes both NOCTI and VTECS tests, in that order, as acceptable measures of student CTE 
skill proficiencies. The state also accepts industry-endorsed competency exams in fields in which 
licensure is required or exams are recognized by the National Skill Standards Board. 
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In Virginia, which bases Per ki ns reporting on student attainment of state-identified CTE 
competencies, the State Board of Education has provided for students to obtain a CTE seal on 
their high school diploma. To earn a seal, students must fulfill the requirements for a standard or 
advanced studies high school diploma, complete a prescribed sequence of courses in a CTE 
program, and either (1) maintain a B or better average in CTE courses, (2) acquire a professional 
license in a CTE field, or (3) pass an exam that confers certification from a recognized industry or 
professional association. As part of this requirement, the state has compiled a list of acceptable 
national industry certification exams (e.g., ASE), as well as endorsed 48 NOCTI occupational 
competency exams. 

New York students who successfully complete a NOCTI or other national industry exam are also 
eligible to earn a technical endorsement on their high school diploma. (Eor Perkins reporting 
purposes, the state reports on the vocational-technical GPA of CTE concentrators.) Since the state 
does not approve, endorse, or certify technical assessments, school district or career technical 
high school administrators are free to select their own assessments from a state -identified pool of 
exams. To assist local districts in identifying qualifying national and state assessments, the state 
has compiled a list of exams and the sponsoring organizations in each trade area. Eor example, 
districts interested in offering a technical endorsement in welding may consult with the American 
Welding Society, New York State Department of Transportation, or NOCTI to identify exams. 

Similarly, in Idaho, which does not have a single statewide assessment, local districts assess 
student skills by selecting their own exams. Districts choose from a variety of options, including 
those produced by NOCTI, as well as developed by other states. The state of Vermont also seeks 
to provide students with access to industry-recognized credentials in areas where credentials 
exist. The state is seeking to identify credentials at the highest levels possible that meet a 
program’ s focus and that stay abreast of industry standard/credentialing options as they develop. 

Einally, conversations with educational administrators in Arizona and Massachusetts indicate 
that these states are in the process of working with NOCTI to identify occupationally specific 
exams that local districts can use to assess student skills. States are still early in their development 
process, however, and currently working to determine a price structure, organization, and funding 
source to structure their testing programs. 

In sum, a small number of states are contracting with NOCTI or adopting credentials developed 
by national industry groups or professional associations to structure their CTE assessments. There 
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are several advantages in using industry-developed assessments for local use. Adopting industry- 
developed exams means that students are assessed for workplace skills that employers’ value, 
ensuring that instruction is aligned with industry needs. States can also avoid upfront 
development cost associated with designing assessments for each of the program areas offered 
within the state, along with the responsibility of having to update exams to keep pace with 
industry developments. Subcontracting assessments can, however, mean that states are forced to 
rely on a third-party agency to determine the content of their CTE assessments, as well as incur 
an ongoing, per student cost in administering and scoring student exams. 

Strategy 2; State Developed, Occupationally Specific, End-of-Course or Program Exams 

Seven states — Kentucky, Mississippi, North Carolina, Ohio, Oklahoma, Utah, and West 
Virginia — ^have established end-of-course or program assessments, aligned with state standards, 
to assess student CTE skill attainment. With some exception, these state-developed tests tend to 
be program specific, meaning that students must demonstrate occupational specific skills 
associated with a discrete occupational area. As might be expected, there is considerable variation 
among states in how these assessments are structured. 

State Assessment Design 

The development of state assessments typically begins with states establishing a core set of CTE 
standards and competencies within a given program area. This process usually entails convening 
a group of state educational and workforce experts, including state and local CTE administrators, 
CTE educators, and business and industry representatives. Committee members are tasked with 
assessing state and national employer needs and surveying existing national standards for 
occupations associated with a given program area. 

Once a set of CTE competencies are identified and validated, committee members, with the 
support of testing experts, begin developing test questions, and in some cases, performance tasks 
that align with identified standards. Initial tests are usually piloted with a subset of schools to 
assess the validity and reliability of exam questions. Skill standards may also serve as a basis for 
developing instructional materials or a statewide curriculum that is aligned to state assessments. 

While creating exams in this way can increase buy-in among educators, drafting state standards 
and assessment can be an expensive process. Given the large number of CTE courses or programs 
that are offered, developing a statewide, occupationally specific testing program can require that 
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states invest substantial resources in identifying and validating the technical skills employers 
desire in a range of programs, and in designing assessments that align with these standards. 
Unfortunately, the literature provides relatively little guidance on the actual cost of developing 
state standards, in part because system development often takes place over many years, and in 
part because not all costs (e.g., task force members’ time) is accurately quantified. 

State End-of-Program Assessment: Kentucky, North Carolina, and Utah 

To illustrate the structure of state testing programs, this section profiles three states — Kentucky 
and Utah — which base assessment on state-developed, end-of-program exams, and North 
Carolina, which structures assessment around state-developed, end-of-course exams. A detailed 
description of these systems is provided in Appendix D. 

States establishing their own testing systems must often create a large number of assessments to 
achieve program coverage. To date. North Carolina has developed roughly 130 end-of-course 
curricular blueprints and assessments, while Utah, which offers end-of-program assessments, has 
established 133 occupationally specific exams. However, not ah states have sought to develop 
assessments for ah occupations: Kentucky has confined assessments to 19 content areas, in most 
cases using one exam to address multiple career majors in a career cluster area. For example, the 
state assessment for the Manufacturing Career Cluster area encompasses ten career majors, 
including welding, machine tool technician, and industrial electronics. 

Unlike states that use NOCTI or other industry-developed exams, states creating their own, 
occupationally specific end-of-course or program exams are responsible for maintaining or 
expanding their CTE assessments. For example, during the 2004-05 program year. North 
Carolina reported validating and determining reliability levels for student assessment measures 
contained in 64 course blueprints, aligning 5 courses with national curricular standards, and 
developed 59 test item banks to assess local educators in developing assessments to prepare 
students for state exams. 

While there is undoubtedly a substantial expense associated with instituting state-specific CTE 
assessments, state profiles suggest that administering statewide testing programs need not be 
prohibitively expensive. For example, Utah estimates that it is able to maintain its annual testing 
program for less than $400,000 a year, with districts providing roughly half of ah resources. 
Federal Perkins funding is used to offset state level expenditures. State administrators in 
Kentucky report budgeting just $80,000 per year to maintain the assessment system, which 
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includes the cost of compensating occupational taskforce partners, producing assessments, 
scoring exams, and reporting performance results. Similarly, administrators in North Carolina 
report that the state spends about $100,000 annually on the testing effort, including development 
of test items, validation and reliability testing, creation of tests, development of manuals used in 
testing, and duplication. 

State-Developed Assessments: Other State Testing Efforts 

Among states employing end-of-program assessments, Oklahoma stands out as a leader in the 
competency-testing arena. Since 1980, the Oklahoma Department of Career and Technology 
Education has worked to (1) identify skills standards that reflect the knowledge and abilities 
needed to perform jobs within an industry, (2) develop curriculum that helps students attain the 
content identified in the skills standards, and (3) create competency assessments that align with 
the skills standards taught using the curricular materials. To assess student performance, the state 
has developed more than 128 occupationally specific, end-of-program tests. These exams consist 
of a performance evaluation and written competency assessment. The state has also recognized 
over 200 alternative assessments, including industry certifications, licensure exams, and tests 
developed by NOCTI, Brainbench, and other agencies, that may be used in place of state- 
developed written competency assessments. 

Ohio also maintains a sophisticated end-of-course and program assessment system to test student 
entry-level occupational knowledge. A total of 48 criterion-referenced occupational assessments 
have been developed, with each exam linked to an Occupational Competency Analysis Profile or 
Integrated Technical and Academic Competency list. End-of-program tests, consisting of roughly 
100 multiple -choice questions, are typically administered to seniors during or after completing 
their last class in a CTE sequence. Modular tests consisting of 20 to 30 multiple-choice items 
associated with a single -unit instructional area, are designed for juniors or seniors following 
course completion. Generally, a program will consist of a sequence of 8 to 10 modules. Exams 
may be administered either on-line or using traditional paper and pencil approaches. Similarly, 
West Virginia provides for end-of-course technical skills tests to assess students’ attainment on 
state content standards and objectives. The state currently provides for 1 17 exam areas. 

The state of Mississippi assesses CTE skill attainment using the Mississippi Career Planning and 
Assessment system. Occupationally specific assessments are administered to all secondary 
students completing a CTE sequence. The state currently maintains 31 program area exams that 
are aligned with CTE curricula, and are revised on the same four-year cycle as the curriculum 
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they are designed to assess. Test item questions are administered as multiple -choice options, with 
students entering their answers on machine scannahle forms. As of the 2005-06 school year, the 
state has discontinued use of ACT’s WorkKeys Workforce Readiness Assessment for CTE 
concentrators, primarily because local districts were not requesting it for their use. 

In sum, states that develop occupationally specific, end-of-course or program exams face the 
prospect of creating and updating a multitude of assessments for different occupations, and in 
some cases multiple skill levels within a single occupation. Annual state investment following 
test development need not he prohibitively expensive, however, with some state maintaining 
testing services at a fraction of that spent in the academic arena. Moreover, since testing systems 
are owned by the state, the marginal cost of assessing students can fall with use. 

Since assessments are usually tailored to address state CTE standards, it is unlikely that most 
occupationally specific assessments can readily transfer across states, although it is difficult to 
assess this without a more detailed comparative study of state systems. It is possible that state 
standards in some occupations may overlap, particularly if a set of nationally recognized 
standards exist and have been consulted in states’ standards and assessment development process. 
Given the amount of work required to develop state standards and assessment systems, states 
would benefit from sharing their existing systems to avoid unnecessarily duplicating effort. 

Strategy 3: State Developed Occupationally Generic, End-of-Program Exams 

Wyoming monitors the skill attainment of CTE concentrators using the Wyoming Career 
Technical Assessment (WyCTA), an electronic, state-developed testing instrument that assess a 
broad set of work readiness skills common to all CTE programs (see Appendix E). Performance 
assessment areas contained within the WyCTA directly link to state content standards that were 
established based on recommendations contained in the 1991 report. What Work Requires of 
Schools: A SCANS Report for American 2000 — published by the Secretary’s Commission on 
Achieving Necessary Skills — and the National Career Development Guidelines, developed by the 
National Occupational Information Coordinating Committee (NOICC) in 1989. 

Wyoming’s CTE standards specify the general skills students are expected to master and perform. 
These standards are not intended to serve as either instructional curricula or technical documents 
to guide day-to-day instruction; rather, teachers are expected to consult the standards when 
developing curriculum. Standards are organized into six major strands: 
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1. Resources: Students manage time, money, materials, facilities and human resources. 

2. Interpersonal Skills: Students acquire and demonstrate interpersonal skills necessary to be 
successful in the workplace. 

3. Information: Students acquire and use workplace information. 

4. Systems: Students demonstrate an understanding of how social, organizational and 
technological systems work. 

5. Technology: Students demonstrate the ability to use a yariety of workplace technologies. 

6. Careers: Students deyelop skills in career planning and workplace readiness. 

The WyCTA proyides performance rubrics coyering six content areas: Communication, Applied 
Math, Affectiye and Thinking, Technology, Pre-Employment, and Employability. These content 
areas align with the content strands, content standards, and benchmarks identified in the state 
standards. Each content area identifies a set of sub-skills that are used for actual rating purposes. 

Unlike most state assessments, which combine a program specific set of multiple -choice 
questions with a performance assessment, the WyCTA is solely performance-based: students are 
rated based on their ability to demonstrate generic skills specified in a set of 18 scenarios. 
Students are rated on their performance by educational staff trained to serye as WyCTA 
eyaluators. Wheneyer possible, CTE instructors are the first choice to serye as raters, since 
eyaluators must be able to obserye students oyer an extended period of time; howeyer, academic 
instructors, guidance counselors, and other school staff may also serye as raters. 

Eyaluators are proyided with a standardized set of rubrics, rubric instructions, and a set of sample 
prompts for each performance scenario. Eyaluators use these rubrics to assign a rating to students 
on each WyCTA sub-skill area, based on obseryations that the eyaluator makes in the classroom 
or other work situation throughout the academic year. In addition to rubrics and prompts, raters 
may consult student portfolios, projects, and other written assessments for use in triangulating 
ratings. Students haye between one to two weeks to complete their assigned problem. All student 
ratings must be completed by April of the current school year. 

Unlike other state assessment systems, the WyCTA proyides educators with an extended 
opportunity to obserye student performance on a task that requires a range of skills. Eor example, 
in one performance scenario, entitled “Cats and Dogs Eyerywhere!” students are asked to solye a 
problem inyolying a local kennel. As part of this one-week assessment, students must identify a 
problem, proyide documentation for the solutions they propose, schedule at least two planning 
sessions that are obseryed by the project rater, demonstrate using a computer to access and use a 
word processing program, deliyer a presentation to a class or small group, and demonstrate 
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listening skills by incorporating presentation feedback into a final written presentation of tbe 
report. Raters evaluate student performance using a set of performance rubrics that specify the 
sub-skills used in the problem, with outcomes rated using on a four-point scale. 

Although the Wyoming assessment system relies on local educators to assess student s ki ll 
attainment, the system is built around a sophisticated set of performance scenarios and assessment 
rubrics that are directly linked to state content standards. Unlike local assessments in other states, 
in which instructors subjectively assess whether students have attained skill proficiencies, 
Wyoming educators are trained in the use of standardized, objective measures of skill attainment. 
Although educators have some flexibility in administering tests (e.g., tailoring prompts to fit local 
conditions or determining timeline for performance). State commissioned studies indicate the test 
provides valid measures of student learning, with high levels of inter-rater reliability. 


According to state administrators, the decision to adopt the WyCTA was conditioned on a 
number of considerations 

1. Annual Expense — To avoid recurring costs associated with administering program specific, 

vendor developed assessments (e.g., WorkKeys), Wyoming established a single exam that 
could be conducted by districts without incurring annual charges. 

2. Instructional Time — Rather than take time out of the instructional day to administer written 
tests, the state opted for a performance -based testing approach that could be incorporated into 
classroom activities. This provided an opportunity for students to learn from their experience, 
while demonstrating their skill mastery. 

3. Local Control — Wyoming is a rural state with a tradition of local control over curriculum and 
assessment. Although the WyCTA prescribes a testing approach, local educators have some 
flexibility in how the test is administered and scored, reducing opposition to the exam. 

4. Authentic Assessment — Given that SCANS focuses on student attainment of many affective 
skills (i.e., thinking and personal qualities, which include responsibility, sociability, self 
management), state administrators believed that a performance -based assessment would 
permit students to demonstrate these skills in a real world context. 

Although Wyoming’s performance-based assessment is intended to evaluate student attainment of 
broad, transferable work readiness skills in a cost-effective manner, it is difficult to quantify the 
actual expense associated with testing. Like most states, Wyoming has invested substantial 
resources to develop its testing instrument; however, the testing program has evolved over time, 
making it difficult to separate out initial development costs from those associated with the final 
assessment. 
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Although annual state expenditures for skill testing are limited to compiling and analyzing testing 
results, local educators invest a suhstantial amount of time administering, evaluating, and 
reporting test ratings. The cost of this instructional time, which may he spread over two or more 
weeks, is difficulty to quantify. 

In sum, Wyoming has taken a unique approach to CTE skill assessment, one that balances state 
needs with federal reporting requirements. Use of a standardized, locally administer performance- 
hased assessment, aligned to a state CTE standards, has enabled state administrators to assess 
student attainment of generic work readiness skills common to all programs. This has helped the 
state to contain assessment costs. Although the state CTE standards and assessments are tailored 
to state needs, Wyoming’s assessment model could be readily transferred to other states. This 
would require, however, that states institute and administer a complex assessment system. Given 
the effort associated with rating individual student performance, the Wyoming approach may be 
more suited to states with smaller CTE student populations. 
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Conclusion 


States are using a variety of approaches to assess secondary students’ CTE skill attainment. 
Among states using some form of standardized skill assessment, roughly half are using vendor- 
developed occupationally specific tests, supplemented with state and industry credentialing 
exams. Remaining states conduct assessments using state developed exams, with most focusing 
on student attainment of occupationally specific skills. Irrespective of testing approach, in most 
cases CTE assessments align with both nationally defined occupational standards and state- 
estahlished CTE performance standards. 

State profiles indicate that states face both start-up and recurring costs in adopting CTE 
assessment systems. Unfortunately, state representatives were unable to fully quantify the cost of 
exam creation in the timeline adopted for this paper. Difficulties in quantifying costs are related 
to the extended time period over which assessments were developed, the number of occupational 
areas selected for assessment, the types of occupations selected, and the difficulty administrators 
have in estimating the cost task force members’ time. It appears that, once developed, states are 
investing between $100,000 to $500,000 annually to maintain their assessment systems. Costs 
vary across states, however, as a function of testing approach, the number of CTE exams used to 
test students, and the number of students participating in exams. 

In most instances, states are using their federal Perkins funding to maintain state testing systems. 
While local districts typically do not have to pay to participate in the assessment system, local 
educators are usually responsible for administering assessments and, in some states, scoring 
performance outcomes. This can require that staff receive specialized training and use 
instructional or other time to conduct assessments. Local educators may also incur some costs in 
duplicating test materials or in transmitting test documents to the state. 

Due to study limitations, it was not possible to compare the relative cost of employing third-party 
assessment systems (i.e., national and industry developed exams) to the cost of using state- 
developed and administered approaches. While more detailed studies of state systems are 
warranted, it would appear, based on conversations with state administrators, that states using 
standardized, vendor- or industry-developed national exams may face somewhat lower start-up 
costs than states developing their own exams. Savings occur because state administrators do not 
need to convene task force members to create and validate CTE standards and assessments in 
each occupational area. 
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Although fiscal data were not available, it would be interesting to assess the marginal cost of 
testing students using third-party versus state-developed assessments. Since states contracting 
with outside vendors face a fixed assessment cost, it may be that state-developed assessments are 
less expensive to conduct in the long run, depending on the number of students tested, the 
lifespan of state exams, and other administrative and scoring factors. Indeed, Pennsylvania 
reports that it spent roughly $540,000 to maintain its statewide testing program in 2005, 
compared to only about $100,000 in North Carolina. 

This statistic takes on meaning when one considers that in the 2003-04 academic year (the most 
recent for which comparable Consolidated Annual Report data are available, Pennsylvania 
reported testing just 16,057 CTE students, compared to 269,147 end-of-course assessments 
administered in North Carolina. This suggests that the per-student cost of assessment in 
Pennsylvania is roughly $33.50, compared to just $0.37 in North Carolina. While there are 
several problems associated with these calculations, a follow-up study to clarify the actual cost of 
assessment using different approaches could surface useful information to assist states in 
selecting a testing strategy. 

Another set of questions relate to whether there are any educational advantages to using national 
versus state-developed assessments. Although conversations with state administrators suggest that 
states using vendor-developed exams are able to tailor assessments to align with state standards, 
whether this approach is as effective as developing state-specific assessments is unknown. 
Conversations with state employers and postsecondary educators could also shed light on whether 
national or state-developed exams confer any advantages over another, as well as whether CTE 
assessments in general, help to prepare students for employment or postsecondary advancement. 

Einally, although most states have developed occupationally specific assessments, some states 
have opted for more generic assessments. Eor example, Kentucky has designed a system of 
assessments that are targeted at the career cluster level, and Wyoming at the general work- 
readiness level. Is one approach superior to another? Or is a combination of work readiness and 
occupationally specific exams desirable? Eurthermore, how do states that use a written, multiple- 
choice exam compare to those that combine a written and performance component? 

While it is beyond the scope of this paper to answer these and other questions, information 
contained within the state profiles section of this report can help states as they begin to prepare to 
adopt new CTE assessment systems. . Initial findings from this paper also suggest that it may be 
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worth developing a mechanism to allow states that have developed their own assessments to share 
these tests with other states. If states lacking CTE assessment systems are to develop their own 
state systems — and additional guidance may he warranted before states proceed in this 
direction — it may he more cost efficient for states to huild off the products developed in other 
states, rather than attempting to design their assessments from the ground up. 
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Appendix A: 

State Data Collection 
Instrument 
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State Profile Data Collection Form 


Assessment History 

When were assessments instituted? 

■ Date instituted 

Why were assessments instituted? 

■ Rationale for assessment 

Development Process 

What was the framework for development? 

■ What process was followed to develop assessments? 

Who participated in desisn/purchase? 

■ Which individuals or groups played a role in assessment development? 

How Ions did it take to develop/institute assessments? 

• What chronology was followed to develop assessments? 

What was the cost of developing state tests? 

■ State investment in developing assessments? 

Have new tests been added over time, and if so, why? 

■ How has the assessment changed over time, if at all? 

■ Why were changes necessary? 

Test Structure 

What types of skills are assessed? 

■ What is the structure of the state exam? 

How many tests exist? 

■ How many program or course assessments currently exist? 

How is local curriculum aligned to state assessments ? 

■ What process was used to align curriculum and assessments? 
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What are the test components and how are they administered? 

■ What are the different test components? 

Are state tests transferable? 

■ Can the state testing program he readily adopted hy other states? 


Test Administration 

Which students take tests? 

■ Who participates in the testing program? 

When are tests administered? 

■ When are tests given? 

Who pays for the assessment? 

■ What entity hears the cost of administration? 

How are testing results used? 

■ What is the purpose of testing? 


State Contact 

■ Name 

■ Title 

■ Phone 
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Appendix B : 

Catalog of NOCTI 
Job Readiness Assessments 
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TABLE OF CONTENTS 

(The number following the test title is for ordering purposes.) 


JOB READY 


Accounting-Basic (2000) 1 

Accounting-Complete (2900) 1 

Administrative Assisting (3001) 1 

Adveitising Design (3019) 1 

Agriculture Mechanics (1002) 1 

Air Cooled Gas Engine Repair (2068) 2 

Appliance Repair (1003) 2 

Architectural Drafting (2004) 2 

Audio-Visual Communications (2005) 2 

Auto Diesel Mechanics (1007) 2 

Automotive Technician Complete (3008) 3 

Automotive Technician Basic (3009) 3 

Building Construction Occupations (1011) 3 

Building Trades Maintenance (2012) 3 

Business Information Processing (2013) 3 

CAD (2973) 4 

CAD/CAM (2073) 4 

Cabinetmaking (1014) 4 

Carpentry (3015) 4 

Clothing and Textiles Management and 

Production (2018) 4 

Collision Repair (2006) 5 

Collision Repair/Refinishing Technology 

(1083) 5 

Commercial Foods (3020) 5 

Computer Networking Fundamentals (2414) 5 

Computer Programming (2023) 5 

Computer Repair Technology (2415) 6 

Computer Technology (3022) 6 

Construction Electi icity (1024) 6 

Construction MasoniY-Block (3025) 6 

Construction Masonry-Brick (3125) 6 

Construction Masoniy-Stone (3225) 7 

Cosmetology (2082) 7 

Criminal Justice-Law Enforcement (2081) 7 

Criminal Justice-General (2981) 7 

Dental Assisting (3026) 7 

Dental Laboratoiy Technology (1417) 8 

Diesel Engine Mechanics (3027) 8 

Eariy Childhood Care & Ed (2016) 8 

Electrical Consbuction (3030) 8 

Electrical Occupations (1031) 9 

Electronic Product Servicing (1078) 9 

Electronic Technology (3035) 9 

Electronics (1034) 9 

Floriculture (3049) 9 

Floriculture-Greenhouse (3949) 10 

Food Production Management and 

Services (3036) 10 

Forestry Products and Processing (2037) 10 

General Drafting and Design (2038) 10 


ASSESSMENTS 


Graphic Communications Techrrology (4042).. 10 

Health Assisting (with Dental) (2043) 11 

Health Assisting (2044) 11 

Heating, Ventilation & Air Conditioning 

(HVAC) (3045) 11 

Heating, Ventilation, Air Corrditioning 

8i Refrigeration (HVAC/R) (3064) 11 

Heavy Equipment Maintenance and 

Repair (2046) 11 

Home Health Aide (2048) 12 

Horticulture-Landscaping (2149) 12 

Horticultur e-Olericultur e (2249) 12 

Hosprtality Mgmt - Food & Beverage (1079)... 12 

Hospitality Mgmt - Lodgirvg (1080) 12 

Industrial Electricity (1050) 13 

Industrial Electronics (2051) 13 

Industrial Maintenance Mechanic (1074) 13 

Manufacturing Technology (1084) 13 

Marine Mechanics (1076) 13 

Medical Assisting (3055) 14 

Metalwor'krng and Fabrication (2056) 14 

Motorcycle Mechanics (1077) 14 

Nursing Assisting (3058) 14 

Painting and Decorating (2060) 14 

Plumbing (3061) 15 

Practical Nursing (3062) 15 

Pre-Engineeting/Engineering Tech (1475) 15 

Pr ecision Machining (3052) 15 

Production Agr iculture (1063) 15 

Refinishing Technology (1075) 15 

Retail Commercial Baking (2010) 16 

Retail Trades (3053) 16 

Robotics Technology (1065) 16 

Technical Drafting (3054) 16 

Television Br oadcasting (1427) 16 

Truck and Bus Mechanics (1069) 17 

Visual Communications (1425) 17 

Warehousing Services (1071) 17 

Welding (3072) 17 

Wor kplace Readiness (3031/3032/3931) 17 
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Appendix C : 

Profiles of States Using 
National Assessment Systems 


Connecticut 

Pennsylvania 
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Connecticut 


Connecticut requires end-of-program testing of all CTE concentrators in each of the 1 8 areas of 
CTE concentration identified hy the State Department of Education for comprehensive high 
schools and correctional institutions. Each assessment has been validated hy individual national 
business, industry and education committees, who ensured that each assessment was both aligned 
with Connecticut’s state defined performance standards and competencies, as well as existing 
national standards in each category of concentration. 


Assessment History 

When were assessments instituted? 

Eollowing two years of research and development, in the 2000-2001 school year the Connecticut 
CTE Assessment program was officially instituted at all Connecticut comprehensive secondary 
schools and correctional institutions. Targeted as the pilot year of the Connecticut CTE 
assessment program, the SDE staff researched and designed a system of testing protocols, 
cyclical communications with high school principals, a multi-layered scoring system, and data 
reporting procedures. 

Twenty high schools were selected to participate in the pilot, representing a cross-section of 
Connecticut's secondary schools in terms of enrollment, urban-suburban-rural. Education 
Reference Groups, and those schools with a broad range of CTE course offerings. The analysis of 
the pilot assessment resulted in substantial changes in our test construction and a number of 
procedural upgrades. 

Why were assessments instituted? 

With the enactment of the Carl D. Perkins III federal legislation, Connecticut initiated research 
and development activities to construct a state assessment program for Career and Technical 
Education that would provide a quality measurement of CTE teaching and learning. 

The Connecticut Performance Standards and Competencies are designed to assist local districts in 
raising expectations, upgrading and improving career and technical curriculum and instruction, 
promoting growth in student achievement, and preparing students effectively for the career and 
technical education statewide assessments. In the State Department of Education’s plan for the 
implementation of statewide CTE assessment, local high schools will determine those students 
who meet the CTE course and competency threshold of being classified as “concentrator.” Such 
classification requires that a student participate in the state assessment process. 


Test Development Process 

What was the framework for development? 

The development of the Connecticut statewide assessment program was initiated with the revision 
of all performance standards and competencies for each of the seven CTE program areas in 1997. 
Research was conducted to identify the Connecticut CTE areas of concentration, and all existing 
national standards for each area. The foundation of the statewide assessment is the testing of only 
measurable competencies. 
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Who participated in desisn/purchase? 

The entire staff of the Career and Technical Education unit of the Bureau of Early Childhood, 
Career and Adult Education assembled representatives from the CTE professional teacher 
organizations, business and industry, and teacher educators for all program areas and identified 
areas of concentration. Representatives of the National Occupational Competency Testing 
Institute (NOCTI) assisted in the initial process of aligning state standards with national standards 
and designed test questions to specifically measure each competency in each area of 
concentration. 

How Ions did it take to develop/institute assessments? 

Extensive CTE research began in 1997, leading to the writing of state performance standards and 
competencies in 18 areas of concentration. Eollowing a year of statewide committee review and 
revision, tests were designed and piloted in 1999-2000. Analysis of the pilot data resulted in the 
final construction of version I of the statewide CTE assessment program that was instituted in the 
spring of 2001. 


What was the cost of developing state tests? 

The approximate cost of the development stage of the Connecticut CTE Assessment Program was 
$300,000. 

Have new tests been added over time, and if so, why? 

Analysis of data from the pilot year and first two years of testing resulted in the discontinuation 
of four of the areas of assessment and the inclusion of three areas. In 2003-2004 Version II of the 
State CTE Assessment program was implemented that included the updated series of written 
assessments for of the seven career and technical education program areas. Each assessment in 
Version II was validated by individual national business, industry and education committees 
whose task it was to ensure that each assessment was aligned with the Connecticut performance 
standards and competencies and all existing national standards. 


Test Structure 

What types of skills are assessed? 

Each area of concentration (test category) is based upon the measurement of the State 
performance standards and competencies. The acquisition of competencies in both CTE and 
academic courses in each of the areas of concentration are measured in the state CTE assessment. 

How many tests exist? 

The Bureau of Early Childhood, Career and Adult Education in partnership with the state career 
and technical education teachers and the National Occupational Competency Testing Institute 
(NOCTI) have developed 18 statewide assessments, covering all the career and technical 
education program areas, based on the Connecticut Standards and Performance Measures. 

The state also collects data on certification tests (CISCO, UNIX, CNA, PLTW, MOUS) that are 
employed by local CTE programs, but these tests represent a very small percentage of students 
and schools. These students (concentrators) are required to take the State CTE Assessment in 
their respective areas of concentration. The state then produces comparative data on the impact of 
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such industry driven training on the state assessment. As an example, there were only 578 
students (concentrators) statewide, participating in industry driven training programs and 
certifications in 2005. Of these students, 45.6 percent were certified nurse assistant concentrators 
and only 314 concentrators were from other certification training programs. This represents only 
3.29 percent of the total concentrators in Connecticut in 2005. 

The focus in Connecticut is clearly on CTE as an impact enhancement for academic achievement 
and a critical feature in directing students to define a four-year college selection and subsequent 
degree. Industry tests as an "offered" means of assessment are rare. The state will he adding at 
least one new area of concentration in this coming year as data (evidence) has clearly defined 
emerging, sustainable areas of concentration in CTE in Connecticut. 

How is local curriculum aliened to state assessments? 

All components of quality Career and Technical Education are connected in Connecticut. Results 
and data from the statewide assessment affect program compliance review, statewide professional 
development and local curriculum revision. All schools receiving Per ki ns grant funding or 
services are required to align local CTE curricular with the state performance standards and 
competencies. 

What are the test components? 

Each tested area of concentration includes a written, multiple -choice examination that addresses 
all competencies contained in that area. The examination period is 90 minutes. 

Performance based assessment was piloted in Connecticut in the Medical Careers program area 
two years ago. Results of the two-year performance based assessment revealed that the process 
had two significant problems. Eirst, the TIME involved to train hundreds of local adjudicators 
was prohibitive. State administrators developed an innovative rubric for performance based 
assessment, lauded by many Connecticut educators, that was designed to make the evaluation of 
student performance as objective as possible, eliminating personal preference/inclination/tradition 
on the part of the industry professional who must evaluate the student. This was a critical piece of 
performance evaluation development. 

The problem with this approach was that to train industry professionals to serve as adjudicators 
took enormous amounts of time. Given that adjudicators are volunteers, time was not a flexible 
factor. The second problem with performance based was the expense involved. As budget options 
were developed, it was easy to see that local schools would have to be willing to absorb some of 
the cost. Given the limited amount of Perkins funds available for most of the state’s secondary 
schools, a mandated performance based assessment package would be difficult to implement. 
The state CTE assessment program is a mandate, not an optional function. Therefore, at this time, 
the state is not planning a performance -based assessment for all our Connecticut high schools. 

Are state tests transferable? 

Research in the areas of concentration suggests that there is not unanimity among states on 
measurable national standards and competencies for CTE program areas and areas of 
concentration. The lack of nationally accepted measurable standards and competencies makes the 
Connecticut CTE statewide assessments unlikely to be easily transferable to other states. The 
process used to develop state assessments may hold promise for replication in other states. 
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Test Administration 


Which students take tests? 

All Connecticut secondary schools and correctional institutions receiving Perkins funds and/or 
services are required to test all students reaching the threshold of concentration. In 2005, 143 high 
schools and correctional institutions participated in the Connecticut CTE Assessment program, 
testing 8,366 concentrators. 

When are tests administered? 

The Connecticut CTE Assessment program is administered during the month of May in all 
schools. 

Who pays for the assessment? 

The state has realized a 14 percent growth in statewide concentrators last year (2005). With the 
increase in concentration, our cost of the state CTE assessment has increased. The cost for the 
State CTE Assessment in 2005 was approximately $265,000. These resources were paid for using 
Carl D. Per ki ns Grant funding. The local school districts (secondary schools and correctional 
institutions) do not pay anything towards the assessment. 

How are testing results used? 

Test results (scored hy performance standard) are promptly (within two weeks of testing-taking) 
returned to each school principal. Schools have the option of using the state assessment results as 
a final examination or the wavier of the final exam. Assessment results from each school are 
analyzed to determine statewide rankings in skill proficiency (1S2) that serves as one of the 
indicators for program compliance review selection. Results are factors in the design of 
professional development and local Perkins grant applications. On a statewide basis, the 
Connecticut CTE statewide assessment data is aggregated and reported to the U.S. Department of 
Education for federal accountahility purposes. 


Contact 

Stephen Hoag, Consultant. State Department of Education, Bureau of Career and Adult 
Education. Phone: 860-713-6764. 
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Pennsylvania 


Pennsylvania uses end-of-program Job Ready Assessments to assess the occupational 
competency of CTE students, with NOCTI exams serving as the primary assessment tool. As of 
January 2005, the state has identified 85 NOCTI Job Ready Assessments and approved 13 
industry-developed assessments for use in assessment. Students who meet the state’s testing 
requirements are awarded a Pennsylvania Skills Certificate. 


Assessment History 

When were assessments instituted? 

In November 1996 the State Board of Education approved an occupational competency measure 
requiring the use of a student occupational competency assessment for all secondary and adult 
career and CTE program completers. . 

Why were assessments instituted? 

Tests were instituted to address federal accountability requirements contained in Perkins and 
because the state is a unified state where state and federal CTE programs are part of the 
Governor’s Executive Order for the Pennsylvania Workforce Investment Board which calls for an 
integrated workforce investment system with core performance measures and standards. Prior to 
these requirements, Pennsylvania had in place a Governor’s Skill Certificate program designed to 
recognize high achievement in students’ chosen occupational field. 


Test Development Process 

Who participated in desisn/purchase? 

NOCTI test development included representatives from industry, secondary and postsecondary 
Education, experts in the field (at least 3 years experience), geographically diverse (at least three 
states represented). The Pennsylvania Department of Education’s (PDE) decision to utilize 
NOCTI Job Ready assessments was made by management in conjunction with the state’s industry 
representatives, career and technical education school administrators and teachers. 

During the process, administrators have learned that there is high value in maintaining constant 
touch with the state’s industry practitioners in revising, updating, benchmarking or even 
customizing tests. Any decisions that are likely to impact workforce training should have input 
from those who will hire program completers. Additionally, consult other states and share with 
them your successes and failures, seeking suggestions on how to improve the assessment system. 

How Ions did it take to develop/institute assessments? 

This was completed over a six-month period. A state team reviewed all existing tests in terms of 
test development, scope, scoring and reporting, reliability, revision schedules, and usability. 
NOCTI was deemed most appropriate for assessment purposes. One of the main reasons for 
choosing NOCTI was that they had a performance component along with the written test. 
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What was the cost of developing state tests? 

The state did not incur any costs associated with developing the tests. Districts select off-the-shelf 
exams for student assessment. 

Have new tests been added over time, and if so, why? 

The state allows districts to select written and performance tests from those available on the 
NOCTI wehsite. Any changes or additions to the NOCTI assessments would affect district exam 
options. 


Test Structure 

What types of skills are assessed? 

NOCTI tests are designed around criteria for joh readiness, meaning that occupationally specific 
skills are assessed. Reported scores are both criterion referenced and norm referenced. 
Pennsylvania is developing statewide cutoff scores to establish a competency standard that 
reflects skills that employers expect from entry-level workers. For sixty-five of the most highly 
used tests, criterion-referenced cut scores are used to determine competency; the remaining tests 
use the NOCTI national norm established at the beginning of the testing cycle. Industry exams are 
occupationally specific. 

How many tests exist? 

Pennsylvania uses five systems of assessment to determine occupational competency of CTE 
students. These include 

1. End-of-program Job Ready Assessments from NOCTI; 

2. End-of-program credentialing tests from industries that have training programs; 

3. State board tests required for licensure to practice in Pennsylvania; 

4. Course specific credentialing tests that cover small portions of a full-approved program; 

5. Locally developed tests that meet state and industry established standards. 

As of January 2005, the state could choose from 85 NOCTI Job Ready Assessments. In addition 
to the NOCTI exam, the state has approved 13 tests that students may take to obtain a 
Pennsylvania Skills Certificate. These tests were developed by the following agencies: 

■ Air Conditioning & Refrigeration Institute Industry 

■ American Culinary Eederation 

■ American Hotel and Lodging Association Educational Institute 

■ American Welding Society 

■ Computer Technology Industry Association 

■ Consumer Electronics Association 

■ National Automotive Technicians Education Eoundation 

■ National Institute for Automotive Service Excellence and the Automotive Youth 
Educational Systems 

■ National Institute for Metalworking Skills — Level I 

■ Nurse Aide Training and Competency Evaluation Program 

■ Pennsylvania Society of Land Surveyors 

■ State Board of Cosmetology Examination 

■ State Board of Licensed Practical Nursing Examination 
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School administrators may also use end-of-program exams not yet approved by PDE, but must 
receive approval prior to testing. 

How is local curriculum aliened to state assessments? 

State approved exams are based on national and/or industry/based standards. While these 
standards frequently parallel local curricula, when they do not, local administrators are 
encouraged to identify gaps between local curriculum and national standards. To assist educators 
in selecting an exam, each of NOCTFs standardized assessments has a critical core competency 
list, which served as the basis of the test’s development. Schools are also required to select 
NOCTI or other PDE approved tests that match local CTE program curriculum content as closely 
as possible. 

What are the test components? 

The NOCTI Job Ready Assessment is composed of two sections — written and performance. The 
written test consists of 150 to 190 multiple -choice questions that can be answered in 90 to 120 
minutes. The performance test typically takes two to three hours to administer. 

Are state tests transferable? 

Since the state testing system uses NOCTI and other industry-based exams, the Pennsylvania 
occupational testing system could readily be transferred to other states, assuming states’ standards 
aligned with NOCTI test items. 


Test Administration 

Which students take tests? 

The PDE requires that all secondary and adult scheduled completers in approved CTE programs 
complete a test if a NOCTI or other PDE approved exam is available. When no occupationally 
specific test is available, students take the Workplace Readiness test - in conjunction with local 
assessments administered by the school as required by individual programs 

When are tests administered? 

Tests are administered at any time in March, April, or May of the school year. Testing must be 
completed by May 31 2005. Schools on a semester basis also test at the end of the first semester 
(December or January). Course specific tests are administered throughout the year as students 
progress through a program. Eor Perkins accountability purposes the accumulated credentials are 
reported for seniors who are scheduled completers. When licensure tests are taken in the summer 
the results are reported in the following school year. 

Who pays for the assessment? 

The PDE pays for the costs of NOCTI test booklets, test scoring, and test reporting services. 
Costs of materials and supplies required for the performance portion of the test are the 
responsibility of the school. Costs for other PDE approved tests are reimbursed up to $25 per test 
administered, with the exception of NIMS, which is fully covered for all scheduled completers in 
the ten NIMS Level I Machining Skill Certifications. 
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In the 2004-2005 program year a total of $540,000 was spent on student occupational 
competency assessment. This amount was spent on NOCTI tests and reports, NIMS testing and 
reports, NOCTI benchmarking costs, reimhursement for other approved tests, and regional 
workshops on occupational testing. Previous years’ costs were slightly lower. Costs for 
occupational testing are part of the education department’ s annual budget, in the Assessment line 
item. 

How are testing results used? 

Program test results are aggregated and reported to the U.S. Department of Education for federal 
accountability purposes. All students who take the NOCTI Job Ready Assessment and score at or 
above the “Advanced Level” on both the written and performance portions receive a 
Pennsylvania Skills Certificate. Students who pass any of the other PDE approved tests are also 
eligible for a Skills Certificate. 


Contact 

Paul Munyofu, Consultant. Pennsylvania Department of Education, Bureau of Career and 
Technical Education. Phone: 717-783-6992. 
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Appendix D : 


Profiles of States Using 
State-Developed, 
Occupationally Specific 
End-of-Course or Program 
Assessment Systems 


Kentucky 
North Carolina 
Utah 
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Kentucky 


Kentucky has instituted a system of end-of-program assessments that are aligned to state CTE 
standards. Kentucky has developed 18 different CTE skill standards assessments that are used to 
students completing one of the 11 career cluster areas adopted hy the state. The state also 
recognizes industry developed certifications developed in some health and information 
technology fields. 


Assessment History 

When were assessments instituted? 

Eollowing the National Education Summit in 1996, the Kentucky Department of Education in 
conjunction with other state agencies, set out to develop CTE standards in high demand 
occupational areas. Manufacturing served as first assessment area, with subsequent career areas 
modeled on this initial work. The resulting Kentucky Occupational Skills Assessment (KOSSA) 
program was piloted in 1999, with 7 state assessments. In the following years, additional 
assessments were piloted and subsequently added to the state assessment system. As of 2005, the 
state had developed a total of 18 CTE assessments. 

Why were assessments instituted? 

Assessments were instituted as a means of supporting the development of a highly skilled state 
workforce, as well as to fulfill the state’s reporting requirement under Perkins. The assessments 
are also intended for used as an overall program improvement tool. 


Development Process 

What was the framework for development? 

Kentucky started the assessment development process by bringing together teachers, business 
partners and postsecondary partners to review existing national standards in high demand career 
areas. Where necessary, representatives also worked to develop new state standards where 
national standards did not exist. After the new state standards were developed, task force 
members worked together to construct and evaluate assessments. 

Who participated in desisn/purchase? 

Participants in the standards development process included teachers, business and industry 
representatives, and postsecondary partners. 

How Ions did it take to develop/institute assessments? 

The development of state assessments has been an ongoing process, with the state expanding 
from 7 assessments in 1999 to the 18 currently in existence. Each year new items are tested and 
the taskforce reconvenes to review the existing item bank for each test. Eor example, there are 
always 100 multiple -choice items on a test. However, only 75 of the items are valid and carry a 
point value. Twenty-five of the questions each year are new and administered on a pilot basis. 

What was the cost of developing state tests? 

The state invested $100,000 in 2000 and 2001. In 2002, due to constraints, the budget was 
reduced to $80,000, where it remains. The budget covers compensating taskforce partners, and 
producing assessments, scoring, and reports. The original budgets from 2000 and 2001 also 
covered the purchasing of scoring equipment. 
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Have new tests been added over time, and if so, why? 

Kentucky started with just 7 assessments in 1999, with tests associated with career clusters that 
had the highest enrollments. In 2000, the state added 3 new tests on a pilot basis. In 2001, the 
existing 10 tests were administered and 8 additional tests piloted bringing the total number of 
state assessments to 18 in 2003. One additional test was piloted in 2003; however, the state is also 
dropping its Housing and Interiors assessment in the 2005-06 school year due to lack of interest 
in schools. Another test may eventually be added for Fashion/Interior Design. 


Test Structure 

What types of skills are assessed? 

State exams are focused on occupational, employability, and academic competencies. The 
occupational skills account for 50 percent of test items, with employability and academic s ki lls 
each accounting for 25 percent of the exam. 

How many tests exist? 

There are a total of 18 KOSSA assessments in the 11 identified career clusters. The state also 
relies on industry-recognized certifications in certain health science and Information Technology 
fields. 

How is local curriculum aliened to state assessments? 

The Kentucky Department of Education has worked with teachers to help them understand the 
importance of aligning their curriculum with state standards. In particular, the state has sponsored 
summer conferences to train teachers in curriculum alignment. 

What are the test components? 

State KOSSA tests include both multiple-choice items and a problem-based scenario. Multiple 
scenarios are presented to the students, who select one that they respond to with a one-to-two 
page essay. The test takes a total of two hours, with one hour devoted to the multiple-choice and 
one to the scenario. Educators may choose to split the test into two parts, administered over more 
than one day. The cut off score for passing the assessment is 70% on both the multiple choice and 
scenario sections. 

Are state tests transferable? 

The certifications that students gain from passing the assessments are endorsed by Kentucky 
businesses only. They have not been endorsed nationally or reviewed by national industry groups, 
in part because most Kentucky students stay and work in Kentucky after graduation. 
Consequently, it is unlikely that state standards and assessments would readily transfer across 
states. 


Test Administration 

Which students take tests? 

All Kentucky students completing or planning to complete a sequence (3 courses) of a career and 
technical education program are required to take KOSSA. Generally, only junior and seniors are 
assessed; however, in some cases sophomores may quality for assessment. Student unable to pass 
the assessment in their junior year are eligible to participate as a senior. Although the assessments 
are voluntary, schools are expected to assess as many students as possible, and to seek to 
maximize their participation rate. 
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When are tests administered? 

During the 2004-05 testing year, the testing window occurred from February 14 to March 11. The 
state has also begun piloting, in the spring of 2005, an online assessment to enable students to 
complete the KOSSA on an interactive website. 

Who pays for the assessment? 

The state pays for all assessments. The only cost to the district is related to shipping completed 
tests back to the state for scoring and reporting. Staff at the state Division of Career and Technical 
Education electronically score the multiple-choice component of the exam. Scoring of the 
performance based constructed response scenario of the assessment is performed by task force 
members in each skill standard area, who use a scoring rubric that indicates performance levels. 
Scoring is double blind, with students required to obtain a score at or above 3 on a 4-point scale. 

How are testing results used? 

Test results are used for Per ki ns accountability purposes. Kentucky measures CTE skill 
proficiency based on the number of students earning a certificate relative to those taking the 
exam. In some instances, industry organizations are endorsing the state standards and calling for 
their members to recognize the value of the state standards and assessments. Students who have 
received certificates have also been featured in industry newsletters and magazines. 


Lessons Learned 

A state official thinks the process Kentucky has been using to implement the assessments has 
been a good one. Their teachers have “grown tremendously” in understanding the importance of 
having current, aligned curriculum. 

She stated it has taken a lot of hours from outside partners to make the assessments possible and 
that it is important to have these partners involved in every step because they help identify the 
standards that Kentucky employers want and are looking for. 

She also stated there are many things they could do to improve their assessments but that it’s not 
possible due to budget restrictions. In particular, they could do more to test the reliability and 
validity of the questions. She said they had thought about outsourcing before but again it is not 
possible due to budget constraints. 


Contact 

Pamela Moore, Consultant. Kentucky Department of Education, Phone: 502-564-3775. 
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North Carolina 


North Carolina assesses occupational skill attainment of CTE students using post assessments 
contained within its state developed VoCATS system. VoCATS is a comprehensive, competency- 
hased, criterion-referenced computerized instructional management system. The system is 
designed to support CTE educators in planning instruction; assessing students before, during, and 
after instruction; evaluating student mastery of competencies; documenting student achievement; 
and providing accountability data. 

Assessment History 

Standardized assessments have been used since 1990 to document student achievement in CTE 
courses. The system went statewide in 1992-93 and was used to provide data about student 
technical attainment required by federal Per ki ns legislation beginning around 1995. The statewide 
assessments are aligned to course blueprints that show exactly what objectives are to be covered. 
School systems also receive hanks of assessment items aligned to the course objectives for use in 
interim testing. 


Why were assessments instituted? 

The VoCATS system was developed to support CTE educators in improving CTE instruction, 
documenting student gains, and demonstrating student competency mastery. 


Development Process 

Who participated in desisn/purchase? 

Teams of North Carolina teachers developed VoCATS materials with input from business and 
industry representatives. To date, hundreds of business representatives and local administrators, 
and thousands of teachers have been involved in development curriculum materials. 

How Ions did it take to develop/institute assessments? 

It takes about three years to develop a course, including the blueprint and the classroom and 
secure item banks. Once developed, courses are revised as needed, at least once every five years. 
During the revision process, there is one “field testing” year when data are not collected for state 
accountability purposes. 

What was the cost of developing state tests? 

Test development is imbedded in the state curriculum development effort and cannot be reported 
separately. 

Have new tests been added over time, and if so, why? 

Existing materials are continuously updated and new materials are developed as needed. Eor 
example, in the 2003-04 school year the state worked to validate and assess the reliability levels 
for student assessment measures contained in 64 course blueprints, aligned 5 courses with 
national curriculum standards, and developed 59 test item hanks. 
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Test Structure 


What types of skills are assessed? 

VoCATS is a competency-based and criterion referenced instructional management system. 
Assessments are aligned with course blueprints that document the skills students need to know 
and be able to do in specific occupational areas. Classroom hanks provide assessment measures 
for both cognitive and performance skills. Statewide post assessments assessed cognitive 
components only. A project is currently underway to incorporate performance measures in the 
state assessments. 

How many tests exist? 

As of Summer 2005, VoCATS curricular blueprints existed for 131 courses. 

How is local curriculum aliened to state assessments? 

Course blueprints lay out the framework of the curriculum for a given course. Included in the 
blueprint are the units of instruction, the core competencies in each unit, and the specific 
objectives for each competency. The blueprint illustrates the recommended sequence of units and 
competencies and the cognitive and performance weight of the objective within the course. 
Teachers use the blueprint to help in the planning of a course of work for the year, preparing daily 
lesson plans, and constructing instructionally valid assessments. 

Teachers may also draw on 116 hanks of assessment items that are distributed electronically, 100 
curriculum guides, and a generation of secured post assessments for courses supported in the 
Programs of Study. Staff development is also available. 

What are the test components? 

Tests are course specific and are designed to assess specific occupational skills taught within a 
course. The state is currently considering adding a performance component to the state 
assessment process. 

Are state tests transferable? 

Classroom assessment banks are available to users in other states. However, these hanks are 
aligned directly with course blueprints, so their use is not appropriate without modification of the 
curriculum. Secure state post assessments are not available outside of North Carolina. 


Test Administration 


Which students take tests? 

All high school CTE teachers use VoCATS as an assessment tool on an ongoing basis for 
pretests, interim tests, unit tests, 9-week tests, midterms, etc. All LEA are using VoCATS- 
designated software and 95 percent of all high schools and many elementary and middle schools 
have computer hardware to run this software. In addition, all CTE students are to take the state 
assessments unless the student has an lEP that calls for alternate testing. In the 2003-04 program 
year, post assessment data were reported for 298,438 of the 388,766 students identified as 
enrolled in CTE courses for which there were end-of-course tests. Currently the state is working 
on ways to increase the number of students for whom data are reported and to collect data for 
students who take alternate tests. 
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When are tests administered? 


Interim tests are administered on an ongoing basis throughout the school year. Post assessments 
are administered at the end of each term. 

Who pays for the assessment? 

Curriculum development occurs at the state level using Perkins funds. Development of tests is 
imbedded in the curriculum development system and is difficult to separate out. A very rough 
estimate is that the state spends about $100,000 annually on the testing effort, including 
development of test items, validation and reliability testing, creation of tests, development of 
manuals used in testing, and electronic duplication. Local systems pay for duplication of hard 
copies of the tests and purchase software necessary for system operation. 

How are testing results used? 

Local school systems use the data to develop strategies to improve student attainment of technical 
competencies. Post assessment results are reported to the federal government for Perkins 
accountability purposes, and to the North Carolina State Board of Education. 


Contact 

Rhonda Welfare, VoCATS Consultant. North Carolina Department of Public Instruction- 
Secondary Education. Phone: 919-807-3876. 
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Utah 


Utah uses state -developed, end-of-program assessments to assess the occupational competency of 
CTE concentrators. In some program areas, end-of course assessments are also availahle. 


Assessment History 

When were assessments instituted? 

In the late 1980s, the Utah Department of Education piloted competency testing in limited 
courses in Business. These were production tests. 

In 1995, the Utah State Legislature determined that up to 20 percent of Applied Technology 
funding would he allocated based on student performance. In 1995 the state recognized, where 
appropriate, national exams and began developing end-of-course tests with performance 
assessment done in the classroom and a state objective multiple-choice test. Since that time, 
competency based testing has become an integral part of the state’s CTE curriculum. As part of 
the 1995 initiative, 10 percent of CTE funding was earmarked for distribution though the state 
Skill Certificate program. In 1997, this percentage increased to 12 percent for incentive funding. 

Why were assessments instituted? 

The primary purpose of the state’s assessment system is to improve CTE program curriculum and 
instruction though the use of performance -based incentives. The Skill Certificate program is also 
used to report CTE skill attainment for the federal Perkins accountability system. 


Development Process 

Who participated in desisn/purchase? 

State CTE administrators have taken the lead in developing statewide assessments. They have 
been assisted by testing consultants, district directors, and a state committee that provides 
oversight and guidance. 

How Ions did it take to develop/institute assessments? 

In 1995 state CTE administrators approached the State Legislature and requested additional 
funding for CTE; legislators agreed to provide additional funding, but tied resources to 
performance. In particular, the legislature stipulated that the state implement a CTE testing 
program and use the results to award up to 20 percent of Applied Technology Education funding 
based on student performance. Over time, competency based funding has grown to become an 
integral part of the CTE curriculum. . 

A first step in the test development process entailed identifying state CTE standards for each 
program area. To do so, the state consulted national standards and consulted with industry 
representatives and teachers to identify a set of objective statewide standards. 
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What was the cost of developing state tests? 

Data on the initial cost of test development is not available. The state has used teacher teams to 
develop exams, and test item revision is an ongoing process. 


Have new tests been added over time, and if so, why? 

Assessments have been added over time to address new areas. The state also conducts annual test- 
item analysis, revising tests as necessary. 

The state also pilot questions that are not counted in the student’s score, but are used to 
continually update the test item data bank from which the multiple -choice questions are drawn. 


Test Structure 

What types of skills are assessed? 

State CTE Skill Certification Tests are organized by program area, around a distinct set of 
occupationally specific skills. In some program areas, multiple assessments have been developed 
to assess students with different levels of skill. For example, the state offers three assessments in 
the Drafting program area, corresponding to beginning, intermediate, and advanced skills. 

How many tests exist? 

Tests are offered in all CTE program areas, including Agricultural Education, Business 
Education, Family and Consumer Sciences, Health Science & Technology Education, Marketing 
Education, Technology Education, Trade & Technical, and Information Technology. As of 2005, 
the state had developed 133 state exams and identified 25 national industry developed and 
licensure certification exams. 

How is local curriculum aliened to state assessments? 

Standards and objectives have been developed for each test in the assessment system. Standards 
are broad statements by category of knowledge and skills that describe what students should be 
able to do. Objectives are sub-standards or competencies included within a standard. Test 
questions on both the multiple choice and performance assessments are written to measure the 
attainment of an objective within a standard. 


What are the test components? 

Students must pass both a performance assessment and a multiple -choice exam to qualify for a 
certificate. This aspect of combining the performance and objective performance has been in 
place for about five years. Before this time each program area had its own criteria. 

Performance assessments are administered in class and evaluated by a student’s teacher prior to 
the multiple -choice exam. All performance objectives must be completed and evaluated prior to 
the written test. To pass the performance exam, student must achieve a level of ‘moderately’ to 
‘highly’ skilled on all performance objectives included in the performance assessment. Each 
element within the performance assessment is tied to a standard and objective from the course. 
Students may take the performance assessment at any time during the course, and may repeat the 
objectives until they are judged to perform at the required level. 
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Skill Certificate program tests are multiple -choice, consisting of up to 80 questions with four 
possible answer choices. Each test item is linked to a single standard and objective, which is used 
for scoring, reporting and program improvement purposes. While each question is counted 
equally for scoring purposes, particular standards may be weighted more highly though the use of 
additional questions tied to that standard. 


Are state tests transferable? 

Since state assessments align with the state’s CTE performance standards and competencies, it is 
unlikely that the state exam can be readily transferred to other states. 


Test Administration 

Which students take tests? 

Participation in the Skill Certification program is voluntary; however, all districts within the state 
participate in the program. This is due, in part, to the incentive funding tied to student 
performance: districts that achieve higher success rates can qualify for additional funding. In the 
2003-04 school year, over 165,00 students participated in the assessment program. 

When are tests administered? 

Tests are designed to be taken at the end of a course and are administered at the end of each 
trimester or semester. The state has a one to two day turnaround time on scoring. 

Who pays for the assessment? 

The Utah State Office of Education developed tests that are provided to students free-of-charge. 
In some cases, typically involving licensure exams, students may be required to pay for the exam 
and licensure. The state pays for the testing program using Perkins funding and by assessing 
participating districts a fee for services. The annual cost of statewide testing is roughly $400,000 
per year, with half of funds coming from Perkins and the remaining amount from districts. 

How are testing results used? 

Program test results are used to award incentive funding to districts based on the number of 
students who earn a certificate by receiving a substantial score, 80 percent, on both the 
performance and objective tests. Outcomes are also aggregated and reported to the U.S. 
Department of Education for federal accountability purposes. 

Although the assessment system is not a formal certification program, students can use their 
certificates as evidence of their accomplishment when applying for employment or further 
education and training. 


Contact 

Renee Hyer, Skill Certificate Program Specialist. Utah State Office of Education, Career and 
Technical Education. Phone: 801-538-7853. 
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Appendix E: 

Profiles of State Using 
State-Developed, 
End-of-Program Assessments 


Wyoming 
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Wyoming 


Wyoming monitors the skill attainment of CTE concentrators using the Wyoming Career 
Technical Assessment (WyCTA), an electronic, state -developed testing program designed to 
objectively assess whether students master a hroad set of academic and technical work readiness 
skills common to all CTE programs offered in the state. Some scenarios also ask students to 
perform some program specific skills, such as setting up equipment to perform a task. Students 
are assessed based on performance: at the beginning of the testing period students (or groups of 
students) are provided with a scenario that they are asked to solve within a given time period. A 
trained evaluator, who uses state-developed testing rubrics and grading procedures to assess 
student performance, rates student performance. 

Assessment History 


When were assessments instituted? 

The first iteration of the WyCTA was developed in approximately 1992 to respond to the 
accountability provision contained in the 1990 Perkins Act, which required states to develop 
accountability systems for secondary and postsecondary vocational education programs. Due to 
the cost associated with other testing systems, such as ACT’s Work Keys program, the state 
collaborated with the National Center for Research in Vocational Education to develop a 
performance-based assessment. The assessment was revised in 2002 to align with newly 
developed state CTE standards. To facilitate this alignment, the State contracted with V-TECS to 
assist in redesigning the testing rubrics to align with newly identified content standards. 

Why were assessments instituted? 

The assessments were initially instituted to respond to federal Perkins reporting requirements. 
The assessment approach adopted was intended to take into account three issues: 

Cost : To avoid recurring costs associated with test administration of program specific 
assessments (e.g., WorkKeys), the state sought to develop a single exam that could be 
administered by districts without charge. 

Instructional Time : Rather than have to take time out of the instructional day to support testing, 
the state opted for an authentic testing approach that could be incorporated into the school 
day. Students would also have an opportunity to learn from the testing program, rather than 
have to simply respond to questions testing their knowledge. 

Local Control : Wyoming is a rural state with a tradition of local control over curriculum and 
assessment. Although the WyCTA prescribes a testing approach, local educators have some 
flexibility in how it is administered and scored. 


Development Process 


What was the framework for development? 

The original WyCTA was designed to assess skills identified in the influential 1991 SCANS 
report. What Work Requires of Schools: A SCANS Report for American 2000^ published by the 


46 



Secretary’s Commission on Achieving Necessary Skills, as well as recommendations in the 
National Career Development Guidelines, developed hy the National Occupational Information 
Coordinating Committee (NOICC) in 1989. All necessary skills found in the SCANS were 
assessed in the original WyCTA, and continue to he addressed in the revised exam. 

In 2000-2001, the state began a process to develop formal CTE content and performance 
standards. Regional meetings were convened where participants compiled drafts using local 
district standards. The state committee, consisting of regional representatives, then used the 
regional documents to draft the state standards. National standards and several states' standards 
were referenced to establish the rigor of the Wyoming Career/Vocational Education Content and 
Performance. In 2002-2003 the standards were reviewed and revised. The WyCTA was revised 
subsequent to standard identification and adoption. 

Who participated in desisn/purchase? 

Regional groups consisting of representatives from each of the districts participated in WyCTA 
development. Community college, universities, students, and business representatives also 
contributed ideas to the design process. 

How Ions did it take to develop/institute assessments? 

The WyCTA was planed to include training and dissemination that took approximately two 
school years to implement. It has been continuously reviewed and updated as necessary. 

What was the cost of dev elopins state tests? 

Initial costs were related to the development of the aligned rubrics used to develop scenarios. The 
cost in the design and alignment was approximately $20,000. The WyCTA is scheduled for 
upgrade upon reauthorization. 

Have new tests been added over time, and if so, why? 

The WyCTA assessments are not secure, meaning that the state will need to draft additional 
scenarios over time. To maintain the quality of assessments, the state analyzes the internal 
consistency reliabilities (Cronbach’s Alpha) and inter-rater reliabilities of assessment rubrics. 
Validity is assessed in a number of ways: 1) face validity by experts in the area of CTE education 
who helped design and continually improved the assessment; 2) factor analyses demonstrate that 
the assessment areas are measuring the constructs of interest; and 3) concurrent validity was 
established through a study in which the WyCTA results were correlated to the ACT Work Keys 


Test Structure 

What types of skills are assessed? 

State CareerW ocational Standards specify what students must master. They are not instructional 
curricula or technical documents used by teachers to guide day-to-day instruction. Teachers 
ensure that students achieve the career/vocational standards by using a range of instructional 
strategies that they select based on their students' needs. All state standards are identified for 
grades K-4, 5-8, and 9-12 with benchmarks at grades four, eight, and eleven. These standards 
have been organized into six major strands: 
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1. Resources: Students effectively manage time, money, materials, facilities and human 
resources. 

2. Interpersonal Skills: Students acquire and demonstrate interpersonal skills necessary to 
be successful in the workplace. 

3. Information: Students acquire and use workplace information. 

4. Systems: Students demonstrate an understanding of how social, organizational and 
technological systems work. 

5. Technology: Students demonstrate the ability to use a variety of workplace 

technologies. 

6. Careers: Students develop skills in career planning and workplace readiness. 

The strands were modeled after the Secretary’s Commission on Achieving Necessary Skills 
(SCANS) and the National Career Development Guidelines. The graduating class of 2006 will be 
required to meet these as well as all state standards for language arts, math, social studies, 
science, health, physical education, foreign language and fine and performing arts. 

An alignment of performance assessment areas within the WyCTA to the career/vocational 
standards are directly linked to specific content standards areas. The WyCTA is also compatible 
with the performance assessment components of the benchmarks. Assessment alignment with 
state standards is a critical component of state accreditation and future graduation requirements. 

How many tests exist? 

The state has developed a set of 18 scenarios. Each scenario aligns with a different content 
standard and benchmark, and each offers students an opportunity to demonstrate their mastery of 
a skill component and sub-skill area. It is not expected that all of the Performance Assessment 
Criteria will be used to assess all Standards. The scenarios are structured to measure progress 
against the Content and Performance Standards, using the specific Performance Assessment 
Components and Sub-skills that can be linked with the standards. 

How is local curriculum aliened to state assessments? 

Wyoming’s CTE standards specify general skills students are expected to perform. They are not 
intended to serve as either instructional curricula or technical documents to guide day-to-day 
instruction; rather, teachers are expected to help students achieve the state standards by offering 
differentiated instructional strategies selected based on their students' needs. 

The WyCTA is directly aligned to the state CTE standards, and has been updated as changes in 
programming and data collection are made. The exam is also compatible with the performance 
assessment components of the benchmarks. Consequently, local instructors that incorporate the 
skills contained in the state content standards are preparing their students for WyCTA assessment. 

What are the test components and how are they administered? 

The WyCTA consists of performance rubrics covering six content areas: Communication, 
Applied Math, Affective and Thinking, Technology, Pre -Employment, and Employability. Within 
each of these content areas are sub skills for actual rating purposes. 

State assessment is performance based: students are rated based on their ability to demonstrate 
skills specified in a set of statewide rubrics. Students are judged on their performance by 
educational staff who are trained to serve as WyCTA evaluators, and who are able to observe 
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students throughout the school year. Whenever possible, CTE instructors are the first choice to 
serve as raters, although academic instructors, guidance counselors, etc., may also serve as raters. 

Evaluators are provided with a standardized set of rubrics, rubric instructions, and a set of sample 
prompts. Evaluators use these rubrics to assign a rating to students on each WyCTA sub s ki ll 
area, based on observations that the evaluator makes in the classroom or other work situation 
throughout the academic year. Students have between one to two weeks to complete their 
assigned problem. All student ratings using the WyCTA performance rubrics must be completed 
as of April of the current school year. In addition to rubrics and prompts, portfolios, projects, and 
written assessments may be included for use in triangulating ratings. 

Are state tests transferable? 

The WyCTA assessment is a performance -based exam that is directly aligned with state CTE 
standards. Skill assessment focuses on broad, occupationally generic work readiness s ki lls 
common across all program areas. While it is likely these skills can be identified in all CTE 
programs, Wyoming tailored its content standards to reflect state -identified needs. As such, it is 
unlikely that the state’s assessment system could be transferred without substantial changes made 
to the content standards, performance rubrics, and rubric instructions. 


Test Administration 

Which students take tests? 

The WyCTA is administered to all CTE students in 10* grade (i.e., those who have taken at least 
2 semester courses) and all CTE concentrators in the 11* and 12* grades (i.e., those who have 
taken 3 or more semester courses in a vocational program. 

When are tests administered? 

Testing occurs throughout the school year, with administration completed by April. Although 
most scenarios are designed for administration in a one or two-week period, a large project may, 
at the discretion of the instructor, be broken into parts or administered over a longer period of 
time. Alternatively if students work together on a problem as a team, project work may take less 
time than stipulated in the rubric. Scenarios are not intended to be time sensitive. 

Who pays for the assessment? 

There is no charge associated with testing. Local district and school staff administer the 
assessment and enter data into an electronic database. The WDE contracts with PRES Associates 
Inc., located in Jackson, Wyoming, to assess the reliability and validity of the CTE assessment 
and to collect and run the analyses on the raw data that are compiled annually. 

How are testing results used? 

WyCTA assessment scores are obtained from all local education agencies on an annual basis to 
monitor and evaluate instructional activities. The 10* grade data are used for district monitoring 
and for program improvement for students still in school. Data from the 12* grade are used for 
federal reporting and state improvement purposes. 
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Contact 


Mariam Manley, President, Pres Associates. Phone: 301-733-3255. 

Philip Trujillo, Perkins Competitive Grant Consultant. Wyoming Department of Education, 
Career and Distance Education-Data & Technology Services. Phone: 307-777-6808. 
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